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1.  Objective 


Unmanned  vehicles  (UVs)  are  being  used  more  frequently  in  military  operations,  and  the  types 
of  tasks  they  are  being  used  for  are  evolving  in  complexity.  In  the  future  battlefield,  Soldiers  may 
be  given  multiple  tasks  to  perform  concurrently,  such  as  navigating  a  UV  while  conducting 
surveillance,  maintaining  local  security  and  situational  awareness  (SA),  and  communicating  with 
fellow  team  members.  In  order  to  maximize  human  resources,  it  is  desirable  to  designate  a  single 
operator  to  supervise  multiple  UVs  simultaneously.  However,  past  research  has  shown  that 
human  operators  are  often  unable  to  control  multiple  robots/agents  simultaneously  in  an  effective 
and  efficient  manner  {1,  2).  Additionally,  as  the  size  of  the  robot  team  increases,  the  human 
operators  may  fail  to  maintain  adequate  SA  when  their  attention  has  to  switch  constantly  among 
the  robots,  and  their  cognitive  resources  may  be  overwhelmed  by  the  intervention  requests  from 
the  robots  {3,  4).  Wang  et  al.  reviewed  a  number  of  studies  on  supervisory  control  of  multiple 
ground  robots  for  target  detection  tasks  and  concluded  that  “the  Fan-out  plateau  lies  somewhere 
between  4  and  9+  robots  depending  on  the  level  of  robot  autonomy  and  environmental  demands” 
{4,v.  143). 

Research  shows  that  autonomous  cooperation  between  robots  can  aid  the  performance  of  the 
human  operators  (5)  and  enhance  the  overall  human-robot  team  performance  (2).  However,  in 
the  foreseeable  future,  human  operators’  involvement  in  mixed-initiative  teams  will  always  be 
required,  especially  for  critical  decision  making.  Human  operators’  decision  making  maybe 
influenced  by  “implicit  goals”  of  which  the  robots  are  not  aware  (i.e.,  items  that  are  not 
programmed  into  the  behaviors  of  the  robots)  (5).  In  addition,  real-time  development  on  the 
battlefield  may  require  the  human  operator  to  change  the  plan  for  the  robot  team  and/or  for 
individual  robots.  Therefore,  effective  communication  between  the  human  operator  and  the 
robots  is  critical  in  ensuring  mission  successes  (d).  To  achieve  a  better  balance  of  enhancing 
autonomy  and  capability  while  simplifying  human-robot  interaction,  we  developed  an  intelligent 
agent  called  RoboLeader,  a  robotic  surrogate  that  could  help  the  human  operator  coordinate  a 
team  of  ground  robots  (our  FY09  Director’s  Research  Initiative  [DRI])  (7).  In  other  words, 
instead  of  directly  managing  the  robot  team,  the  human  operator  only  dealt  with  RoboLeader; 
consequently,  the  operator  could  better  focus  on  the  other  tasks  requiring  attention.  In  Chen  et  al. 
(7),  the  effectiveness  of  RoboLeader  was  investigated  in  a  human-in-the-loop  simulation 
experiment.  Chen  et  al.  compared  the  operators’  target  detection  performance  in  a  four-  and 
eight-robot  condition.  The  results  showed  that  participants  detected  significantly  fewer  targets 
with  eight  robots  versus  four  robots.  Although  there  were  no  significant  differences  between  the 
RoboLeader  and  baseline  (no  RoboLeader)  conditions  for  target  detection,  the  RoboLeader 
group  reduced  their  mission  completion  times  by  approximately  13%  compared  to  the  baseline 
group.  Those  participants  with  higher  spatial  ability  detected  more  targets  than  did  those  with 
lower  spatial  ability.  Participants  experienced  significantly  higher  workload  with  eight  robots 
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compared  to  the  four-robot  eondition,  and  those  with  better  attentional  eontrol  reported  lower 
workload  than  did  those  with  poorer  attentional  eontrol. 

In  the  eurrent  study,  the  eapabilities  of  RoboLeader  were  expanded  to  deal  more  speeifieally 
with  dynamie  re-tasking  requirements  for  persistent  surveillanee  of  a  simulated  urban 
environment  based  on  various  battlefield  developments  as  well  as  eoordination  between 
unmanned  aerial  vehieles  (UAVs)  and  unmanned  ground  vehieles  (UGVs)  in  pursuit  of  moving 
targets  in  urban  environments.  More  speeifieally,  the  partieipants  used  four  UGVs  to  “ehase”  a 
moving  target  via  a  map  that  displays  the  loeation  of  the  target  via  a  UAV  link  while  monitoring 
the  streaming  videos  from  the  four  UGVs.  The  levels  of  autonomy  of  RoboLeader  were 
manipulated  (see  seetion  2.3).  Additionally,  we  investigated  the  effects  of  individual  differences 
in  spatial  ability  (SpA)  and  pereeived  attentional  control  (PAG)  on  the  operators’  roboties  eontrol 
as  well  as  multitasking  performance. 

Previous  researeh  has  found  performanee  differenees  on  eertain  tasks  between  individuals  with 
high  and  low  SpA.  Lathan  and  Tracey  (5)  demonstrated  that  people  with  higher  SpA  finished 
their  tasks  faster  and  had  fewer  errors  during  a  teleoperation  task  through  a  maze.  Lathan  and 
Traeey  suggested  that  military  missions  ean  benefit  from  seleeting  personnel  with  higher  SpA  to 
operate  robotie  deviees.  Our  previous  studies  also  found  SpA  to  be  a  good  predietor  of  the 
operator’s  roboties  performanee  {1,  7).  In  Chen  et  al.  (7),  partieipants  with  higher  SpA  seanned 
the  videos  signifieantly  faster  than  those  with  lower  SpA.  In  the  eurrent  experiment,  we  also 
examined  the  relationship  between  attentional  eontrol  and  multitasking  performanee.  Several 
studies  show  that  there  are  individual  differenees  in  multitasking  performanee,  and  some  people 
are  less  prone  to  performanee  degradation  during  multitasking  eonditions  (P).  There  is  also 
evidenee  that  people  with  better  attentional  eontrol  ean  alloeate  their  attention  more  flexibly  and 
effeetively  {10),  whieh  was  partially  confirmed  by  Chen  and  Joyner  {11). 

In  the  eurrent  study,  we  manipulated  the  level  of  autonomy  of  RoboLeader  and  examined  its 
effeet  on  the  operator’s  performanee  (i.e.,  plan  revisions  for  the  robots,  the  eoneurrent  target 
deteetion  task,  and  SA  of  the  mission  environment)  and  workload.  The  four  levels  of 
manipulation  were  as  follows:  Manual,  Semi- Autonomous  without  Visualization,  Semi- 
Autonomous  with  Visualization,  and  Fully  Automated.  The  Semi- Autonomous  eondition  was 
divided  into  two  conditions  so  that  the  effect  of  the  visualization  tool  eould  be  evaluated.  The 
visualization  tool  informed  the  partieipant  of  the  synehronization  of  the  robots  as  well  as  overall 
entrapment  effeetiveness  of  the  target  based  on  the  movement  of  the  target  (see  seetion  2.2.2). 
The  effeets  of  operator  individual  differenees  in  spatial  ability  and  attentional  eontrol  were  also 
investigated. 

There  were  two  major  risks  that  needed  to  be  overeome  in  order  for  this  DRI  to  be  suceessful. 
One  was  developing  the  algorithms  for  the  RoboLeader  system  and  its  interaction  with  the 
human  operator  as  well  as  individual  robots  under  its  control.  In  human-robot  interaction 
research,  predetermined  (eanned)  simulations  are  often  used  to  simulate  automated  capabilities. 
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In  this  DRI,  RoboLeader  was  capable  to  provide  real-time  solutions  to  the  human  operator  based 
on  the  operator’s  responses.  Another  risk  involved  in  this  DRI  was  developing  the  visualization 
tool.  While  visualization  teehniques  have  been  researched  for  a  long  time,  not  much  research  has 
been  eonducted  in  the  area  of  providing  real-time  visual  feedbaek  to  operators  about  their 
planning  task. 


2.  Approach 

2.1  Participants 

Twenty-eight  individuals  in  the  Orlando,  FL,  area  (3  females  and  25  males,  mean  age  21  years) 
participated  in  this  study.  They  were  eompensated  $15  per  hour  for  their  partieipation. 

2.2  Apparatus 

2.2.1  Simulator 

The  Mixed  Initiative  Experimental  (MIX)  Testbed  used  in  the  first  RoboLeader  experiment  was 
modified  and  used  as  the  simulator  for  this  experiment  (7).  The  MIX  Testbed  is  a  distributed 
simulation  environment  for  investigating  how  unmanned  systems  are  used  and  how  automation 
affeets  human  operator  performanee.  The  Operator  Control  Unit  (OCU)  of  the  MIX  Testbed  was 
modeled  after  the  Taetical  Control  Unit  developed  under  the  U.S.  Army  Research  Laboratory 
(ARL)  Robotics  Collaborative  Technology  Alliance.  This  platform  ineludes  a  camera  payload 
and  supports  multiple  levels  of  automation.  Users  ean  send  mission  plans  or  teleoperate  the 
platform  with  a  computer  mouse  while  being  provided  a  video  feed  from  the  eamera  payload. 
Typieal  tasks  inelude  reeonnaissanee  and  surveillanee.  RoboLeader  is  eapable  of  eolleeting 
information  from  subordinate  robots  with  limited  autonomy  (e.g.,  with  eapability  of  eollision 
avoidanee  and  self-guidanee  to  reaeh  target  loeations),  making  tactieal  decisions,  and 
coordinating  the  robots  by  issuing  eommands,  waypoints,  or  motion  trajeetories  (7). 

2.2.2  RoboLeader 

The  RoboLeader  user  interfaee  is  shown  in  figure  1 .  The  map  was  loeated  in  the  upper-left 
eorner  and  showed  the  loeation  of  the  moving  target  (a  slowly  moving  truck)  via  a  link  to  a  UAV 
flying  over  the  mission  area  (i.e.,  the  ieon  representing  the  truek  was  displayed  as  an  arrow  to 
show  the  direetion  of  movement  and  eontinuously  moved  on  the  map  display  based  on  the 
truek’s  actual  movement,  whieh  was  updated  from  the  UAV).  The  right  half  of  the  sereen  was 
divided  into  four  separate  eamera  feeds,  streaming  from  the  four  UGVs.  The  visualization 
display  was  loeated  in  the  lower  left  eorner  of  the  OCU.  The  four  vertieal  bars  had  the  words, 
“NEAR”  and  “EAR”  at  the  top  and  bottom  of  eaeh  bar,  respeetively,  and  represented  the  seores 
for  eaeh  robot’s  synehronization  with  the  moving  target;  the  horizontal  bar  represented  the 
aggregate  entrapment  seore  of  the  robots’  plans.  As  the  target  moved,  RoboLeader  ealeulated  the 
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scores  based  on  eaeh  robot’s  plan  as  well  as  the  speed  and  movement  of  the  target  and  showed 
the  scores  via  the  bar  graphs.  The  vertical  bars  filled  with  the  color  green  as  a  specific  UGV  got 
closer  to  the  target  or  filled  with  red  as  a  UGV  got  farther  from  the  target.  Additionally,  a  red 
arrow  was  displayed  to  the  right  of  eaeh  graph  if  a  UGV  was  moving  away  from  the  target  (see 
figure  1).  RoboLeader  continuously  updated  the  calculations  and  visualizations  (bar  graphs)  until 
the  target  was  eaptured  (cornered  by  at  least  two  UGVs),  at  whieh  time  the  seenario  ended. 


Figure  1 .  RoboLeader  operator  control  unit. 

The  primary  driver  algorithm  implemented  for  RoboLeader  was  based  on  the  A-Starr  algorithm 
and  is  a  one-step  seareh  routine  {12).  In  its  basie  form,  the  design  was  used  to  effieiently  track 
between  two  way  points:  a  starting  point  and  an  ending  point,  where  the  starting  point  was 
typically  the  eurrent  position  of  an  intereept  vehiele  and  the  ending  point  was  a  designated  target 
intereept  point.  Onee  a  target  location  and  heading  was  identified  within  the  terrain,  RoboLeader 
determined  all  possible  approach  routes  to  the  target  and  then  determined  which  vehicle  should 
traek  to  a  partieular  intereept  loeation. 

Figure  2  depiets  the  layout  of  a  target  within  an  urban  street  terrain.  In  this  partieular  ease,  four 
street  interseetions  eomprise  possible  approaeh  routes  for  the  intereept  vehieles  whereas  three 
intersections  are  of  primary  concern  in  order  to  block  possible  escape  routes.  In  order  to  fully 
encapsulate  a  target,  however,  intereept  vehicles  must  be  dispatched  to  the  primary  intercept 
routes  first  and  then  any  remaining  vehiele  assets  can  then  be  assigned  to  fall  in  behind  the  target 
in  a  tail-ehase  fashion.  Once  all  target  intercept  points  are  identified,  every  possible  path  between 
eaeh  vehiele  and  target  intereept  points  is  ealeulated  and  logged  into  temporary  memory.  In  other 
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words,  for  each  vehicle,  a  path  is  calculated  from  that  vehicle  to  each  of  the  target  intercept 
points,  which  results  in  n-t  calculated  paths,  where  n  is  the  number  of  intercept  vehicles  and  t 
is  the  number  of  target  intercept  routes.  RoboLeader  calculates  the  length  of  each  of  these  paths 
and  stores  this  data  into  temporary  memory  as  well. 


Figure  2.  Target  intercept  geometry. 

In  order  to  determine  which  vehicle  gets  assigned  to  particular  target  approach  point, 
RoboLeader  analyzes  the  collected  path  data  and  specially  arranges  that  data  with  the  help  of 
C++  sort  predicates.  RoboLeader  then  identifies  the  vehicle  with  the  closest  approach  route  to 
the  target  and  commands  that  vehicle  to  navigate  to  the  appropriate  intercept  point.  This  process 
continues,  giving  the  next  closest  vehicle  precedence  until  all  assets  are  allocated.  Once  nested 
within  the  real-time  environment  of  the  Mix  Testbed,  the  target  tracking  and  vehicle  allocation 
process  occurred  iteratively  at  a  set  rate.  Therefore,  each  time  the  RoboLeader  algorithm  was 
called,  fresh  new  target  data  were  gathered  and  target  intercept  trajectories  calculated.  This 
ensured  constantly  updated  target  entrapment  routes  in  real  time. 

In  addition  to  real-time  target  entrapment,  RoboLeader  also  provided  real-time  assessment  of 
participants’  target  entrapment  performance,  which  was  displayed  in  the  visualization 
components  of  the  RoboLeader  user  interface.  In  the  two  conditions  when  the  visualization  tool 
was  unavailable  (Manual  and  Semi- Autonomous  without  Visualization),  RoboLeader  calculated 
the  encapsulations  scores  (for  data  analysis  purposes)  but  did  not  provide  feedback  to  the 
operators.  There  were  two  levels  of  feedback  on  target  tracking  and  entrapment: 

•  Directional  Metric.  The  first  metric  was  a  directional  indicator  and  was  expressed  with 
respect  to  relative  headings  between  the  intercept  vehicle  and  the  target.  Because  vehicle 
motions  were  constrained  within  an  urban  setting,  it  was  important  to  know  whether  the 
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overall  motion  of  the  intereept  vehiele  was  trending  toward,  parallel  to,  or  away  from  the 
target.  Establishment  of  a  line  of  sight  (LOS)  between  eaeh  of  the  intercept  vehicles  and  the 
target  was  important  in  determining  the  relative  directional  trend.  To  do  this,  a  simple 
vector  projection  equation  was  used  that  projected  the  intercept  vehicles  heading  vector 
onto  the  LOS.  The  separation  angle  between  the  LOS  and  heading  vector  was  then 
calculated.  Ligure  3  shows  how  the  metric  was  formed.  V  represents  the  intercept  vehicles 
heading  with  respect  to  the  LOS. 


V  ^  ^ 

V 

\ 

V 
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'  'LOS 

'  LOS 

'  LOS 
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'  'LOS 
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Figure  3.  Directional  metric  calculation. 

The  angle  6  range  is  (^,0)  whereas  the  cosine  of  the  angles  range  is  -1  <  cos0  <  +1  and 
therefore,  the  calibration  of  the  directional  indicator  is  scaled  directly  to  the  cosine  of  the 
angle  where  -100%  indicates  that  the  intercept  vehicle  is  moving  in  the  opposite  direction 
to  the  target  and  +100%  indicates  that  the  intercept  vehicle  is  moving  directly  at  the  target. 

•  Entrapment  Metric.  The  second  metric  measured  how  well  the  target  escape  routes  had 
been  blocked  by  intercept  routes.  As  discussed  previously,  a  target  could  only  have  a  set 
number  of  escape  paths  from  its  current  position.  The  RoboLeader  algorithm  determined 
these  escape  routes  and  assigned  intercept  vehicles  to  these  points  in  order  to  make  an 
intercept.  The  entrapment  score  was  simply  based  on  how  many  of  these  escape  routes  had 
an  intercept  vehicle  assigned  to  them. 

2,2,3  Surveys  and  Tests 

A  demographics  questionnaire  was  administered  at  the  beginning  of  the  training  session.  An 
Ishihara  Color  Vision  Test  (with  nine  test  plates)  was  administered  via  PowerPoint® 
presentation.  Since  the  RoboLeader  OCU  employed  several  colors  to  display  the  plans  for  the 
robots,  normal  color  vision  was  required  to  effectively  interact  with  the  system.  A  questionnaire 
on  Attentional  Control  {10)  was  used  to  evaluate  participants’  perceived  attentional  control.  The 
Attentional  Control  survey  consists  of  20  items  and  measures  attention  focus  and  shifting.  The 
scale  has  been  shown  to  have  good  internal  reliability  (a  =  0.88).  The  Cube  Comparison  Test 
{13)  and  the  Spatial  Orientation  Test  {14)  were  used  to  assess  participants’  spatial  ability. 
According  to  Neumann  {15),  these  two  tests  measure  different  aspects  of  spatial  ability.  The 
Cube  Comparison  Test  requires  participants  to  compare,  in  3-min,  21  pairs  of  six-sided  cubes 
and  determine  if  the  rotated  cubes  are  the  same  or  different.  The  Spatial  Orientation  Test, 
modeled  after  the  cardinal  direction  test  developed  by  Gugerty  and  his  colleagues  {14),  is  a 
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computerized  test  consisting  of  a  brief  training  segment  and  32  test  questions.  Both  aecuracy  and 
response  time  were  automatieally  eaptured  by  the  program. 

Partieipants’  pereeived  workload  was  evaluated  with  the  eomputerized  version  of  the  National 
Aeronauties  and  Spaee  Administration  Task  Load  Index  (NASA-TLX)  questionnaire,  whieh 
used  a  pairwise  comparison  weighting  proeedure  {16).  The  NASA-TLX  is  a  self-reported 
questionnaire  of  pereeived  demands  in  six  areas:  mental,  physieal,  temporal,  effort  (mental  and 
physieal),  frustration,  and  performanee.  Partieipants  evaluate  their  pereeived  workload  level  in 
these  areas  on  10-point  seales  as  well  as  eomplete  pairwise  eomparisons  for  eaeh  subseale.  A 
modified  version  of  the  Usability  survey  used  in  Chen  and  Terrenee  (77)  was  used  to  assess 
partieipants’  pereeived  usability  of  the  RoboLeader  system  as  well  their  trust  in  the  system.  The 
items  that  assessed  participants’  trust  in  the  system  were  modified  from  the  “Trust  Between 
People  and  Automation”  survey  {18). 

2,3  Procedure 

After  being  briefed  on  the  purpose  of  the  study  and  signing  the  informed  consent  form, 
partieipants  eompleted  the  demographies  questionnaire  and  the  Attentional  Control  survey,  and 
were  administered  a  brief  Ishihara  Color  Vision  Test  to  ensure  they  had  normal  eolor  vision. 
After  the  Color  Vision  Test,  the  partieipants  eompleted  the  two  spatial  ability  tests.  Partieipants 
then  reeeived  training  and  praetieed  on  the  tasks  they  would  need  to  eonduet  during  the 
experimental  session.  Training  was  self-paeed  and  was  delivered  by  PowerPoint  slides  showing 
the  elements  of  the  OCU,  steps  for  completing  various  tasks,  several  mini-exereises  for 
praetieing  the  steps,  and  exereises  for  performing  the  robotie  eontrol  tasks.  The  training  session 
lasted  approximately  1  h. 

The  experimental  session  lasted  about  1.5  h  and  began  immediately  after  the  training  session. 
Each  experimental  session  had  four  seenarios  (eorresponding  to  the  four  experimental 
eonditions;  however,  the  pairing  of  seenarios  and  eonditions  was  eounterbalaneed),  eaeh  lasting 
approximately  20  min.  During  the  seenarios,  partieipants  used  their  four  robotie  assets  to  pursue 
a  primary  moving  target  (a  truek  traveling  at  about  3  MPH)  while  monitoring  the  streaming 
videos  from  the  robots  in  order  to  find  additional  (seeondary)  targets  (insurgents  earrying 
weapons)  in  the  mission  environment.  When  the  seenario  for  the  Manual  eondition  started,  the 
partieipants  put  in  waypoints  for  each  UGV  manually  and  adjusted  the  waypoints  based  on  the 
movement  of  the  primary  target.  In  the  Semi- Autonomous  eonditions,  the  partieipant  seleeted  an 
end  point/loeation  for  the  UGV  at  whieh  time  RoboLeader  provided  an  optimum  solution  with 
how  to  reach  the  desired  destination.  With  the  visualization  eondition,  the  user  might  eonsult  the 
bar  graphs  as  an  indieator  of  whether  their  point  seleetions  were  effeetive  in  terms  of 
synehronization  of  the  robots  and  entrapment  of  the  target  or  if  the  plans  needed  revisions.  The 
scores  displayed  in  the  visualization  area  were  ealculated  based  on  the  RoboLeader’s 
eneapsulation  algorithm.  Without  visualization,  the  partieipant  had  to  determine  if  they  were 
properly  eomering  the  target  for  eapture.  In  the  Fully  Automated  eondition,  RoboLeader 
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provided  the  recommended  end  points  as  well  as  intermediate  waypoints  for  each  robot.  The 
participant  could  accept,  modify,  or  reject  the  plans.  In  each  scenario,  there  were  hostile  areas 
(indicated  by  red  squares  on  the  map)  that  the  robots  needed  to  avoid.  The  order  of  experimental 
conditions  was  counterbalanced  across  participants. 

The  robots  did  not  have  aided  target  recognition  capability;  therefore,  the  participants  had  to 
detect  the  insurgents  (secondary  targets)  by  themselves.  For  the  insurgent  targets,  participants 
used  a  mouse  to  click  the  Insurgent  button  on  the  interface  and  then  click  on  the  insurgent  to 
“laze”  them  as  soon  as  they  were  detected.  The  “lazed”  insurgent  was  then  displayed  on  the  map. 
Additionally,  there  were  civilians  as  well  as  friendly  Soldiers  in  the  simulated  environment  to 
increase  the  visual  noise  present  in  the  target  detection  tasks. 

Each  scenario  also  contained  five  SA  queries,  which  were  triggered  based  on  time  progression 
(e.g.,  3  min  into  the  scenario).  The  SA  queries  included  questions  such  as  “Use  the  provided 
paper  to  identify  which  route  or  routes  have  encountered  the  most  Insurgents,”  etc.  When  an  SA 
query  was  triggered,  the  OCU  screen  went  blank,  the  simulation  was  paused,  and  the  SA  query 
was  displayed  on  the  screen.  Participants  then  wrote  their  response  to  the  query  on  an  answer 
sheet.  After  participants  responded  to  the  SA  query,  it  was  removed  from  the  OCU  screen  and 
the  simulation  resumed. 

There  was  a  2-min  break  between  the  experimental  scenarios.  Participants  assessed  their 
perceived  workload  using  the  NASA-TLX  as  well  as  their  perceived  usability  of  the  RoboLeader 
user  interface  using  the  Usability/Trust  survey  immediately  after  each  experimental  scenario. 

The  entire  data  collection  session  lasted  about  3  h. 

2,4  Experimental  Design  and  Performance  Measures 

The  study  was  a  within-subject  design  with  RoboLeader’s  level  of  autonomy  as  the  independent 
variable  (with  four  levels:  Manual,  Semi- Autonomous  without  Visualization,  Semi- Autonomous 
with  Visualization,  and  Fully  Automated).  Dependent  measures  included  the  participants’ 
performance  of  encapsulating  the  primary  target  (the  encapsulation  scores),  the  percentage  of 
secondary  targets  (insurgents)  detected,  the  participants’  SA  of  the  mission  environment 
(percentage  of  SA  queries  answered  correctly),  and  the  participants’  perceived  workload.  A 
repeated-measure  analysis  of  variance  with  RoboLeader  as  the  within-subject  factor  was  used  to 
evaluate  the  operator  performance  differences  among  the  four  conditions. 
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3.  Results 


3,1  Primary  Target  Encapsulation  Performance 


The  analysis  revealed  that  the  level  of  autonomy  of  RoboLeader  significantly  affected  the 
participants’  performance  of  primary  target  encapsulation,  F(3,24)  =  \0.6\,p  <  0.0001 
(figure  4).  Post  hoc  comparisons  (least  significant  difference  [LSD])  showed  that  the 
participants’  encapsulation  scores  in  the  Fully  Automated  condition  were  significantly  better 
than  their  results  in  the  other  conditions.  On  average,  63%  of  the  participants  successfully 
encapsulated  the  primary  target  in  the  Manual  condition;  96%  of  them  did  so  in  the  Semi- 
Autonomous  without  Visualization  condition;  86%  in  the  Semi- Autonomous  with  Visualization 
condition  and  the  Fully  Automated  condition  (figure  5).  A  y  test  showed  that  the  percentage  of 
perfect  encapsulation  in  the  Manual  condition  was  significantly  lower  than  the  average  of  the 
other  three  conditions  (i.e.,  RoboLeader-assisted  conditions),  y  (1,  A^=  27)  =  1 1.901, <  0.001. 
In  the  Manual  condition,  participants  who  play  video  games  frequently  (i.e.,  daily  or  weekly) 
performed  significantly  better  than  infrequent  gamers  y  (1,  A=  7)  =  3.857, <  0.05.  On 
average,  frequent  gamers  successfully  encapsulated  the  target  70%  of  the  times  and  infrequent 
gamers  did  so  only  43%  of  the  times.  Participants’  spatial  ability  and  perceived  attentional 
control  did  not  have  effects  on  their  encapsulation  performance. 


Figure  4.  Encapsulation  performance. 
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Figure  5.  Percentage  of  participants  with  successful  encapsulation  performance. 


3,2  Secondary  Target  Detection  Performance 

On  average,  the  participants  detected  approximately  two  thirds  (61-64%)  of  the  insurgents  in  the 
mission  environment.  However,  there  was  not  a  significant  main  effect  of  RoboLeader’s  level  of 
autonomy  on  participants’  target  detection  performance.  Neither  of  the  individual  difference 
factors  (participants’  spatial  ability  and  perceived  attentional  control)  had  a  significant  effect  on 
the  target  detection  performance. 


3,3  Situational  Awareness 


The  participants’  SA  of  the  mission  environment  did  not  differ  among  the  conditions. 
Participants’  gaming  experience  was  found  to  have  an  effect  on  their  SA,  F(2,24)  =  3.476, 
p  <  0.05  (figure  6).  Post  hoc  comparisons  (LSD)  showed  that  the  daily  gamers  and  weekly 
gamers  outperformed  infrequent  gamers,/?  <  0.05.  There  was  no  difference  between  daily  and 
weekly  gamers. 
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Figure  6.  Effects  of  gaming  experience  on  SA  of  the  mission  environment. 
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3,4  Workload 

There  was  not  a  significant  main  effect  of  RoboLeader’s  level  of  autonomy  on  participants’ 
perceived  workload  (weighted  NASA  TLX  composite  scores;  figure  7).  However,  when  the 
workload  assessment  in  the  Manual  condition  was  compared  against  the  aggregated  workload 
assessments  of  the  other  three  conditions  combined,  there  was  a  significant  difference  between 
the  Manual  and  the  RoboLeader-assisted  conditions,  F(l,25)  =  8.312,/>  <  0.05.  Participants 
reported  significantly  higher  workload  ratings  in  the  Manual  condition  (M=  63.87)  than  they  did 
in  the  other  three  conditions  (M=  58.36). 
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Figure  7.  Perceived  workload. 


4.  Conclusions 


In  the  FY09  DRI,  we  developed  the  RoboLeader  intelligent  agent  that  had  the  capabilities  of 
coordinating  a  team  of  ground  robots  and  revising  route  plans  for  the  robots  based  on  battlefield 
intelligence  (7).  In  the  current  DRI,  the  capabilities  of  RoboLeader  were  expanded  to  deal  more 
specifically  with  dynamic  re-tasking  requirements  based  on  battlefield  developments  as  well  as 
coordination  between  UAVs  and  UGVs  in  pursuit  of  moving  targets.  The  results  of  our  human- 
in-the-loop  simulation  experiment  showed  that  RoboLeader  (Fully  Automated  condition)  was 
more  effective  in  encapsulating  the  moving  targets  than  were  the  human  operators  (when  they 
were  either  without  assistance  from  RoboLeader  [Manual]  or  when  they  were  partially  assisted 
by  RoboLeader  [Semi- Autonomous]).  Participants  successfully  encapsulated  the  moving  targets 
only  63%  of  the  time  in  the  Manual  condition  but  89%  of  the  time  when  they  were  assisted  by 
RoboLeader  (i.e.,  the  other  three  conditions  combined).  Those  participants  who  play  video 
games  frequently  (daily  or  weekly)  demonstrated  significantly  better  encapsulation  performance 
than  did  infrequent  gamers;  they  also  had  better  SA  of  mission  environment.  These  results  are 
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consistent  with  the  findings  of  one  reeent  U.S.  Air  Foree  study  (20)  that  frequent  video  gamers 
outperformed  infrequent  gamers  on  robotics  (unmanned  aerial  systems  [UAS])  tasks  and,  in 
some  cases,  performed  as  well  as  experienced  pilots.  It  is  somewhat  surprising  that  visualization 
had  little  effect  on  the  Semi- Autonomous  eonditions.  This  suggests  that  map  cues  were  sufficient 
for  the  eneapsulation  task  in  the  eurrent  experiment.  It  is  worth  noting  that  in  the  eurrent  study, 
the  speed  of  the  moving  target  was  fairly  slow  (about  3  MPH),  which  made  the  target 
eneapsulation  task  relatively  easy.  If  the  moving  target  had  traveled  at  a  higher  speed,  the  target 
eneapsulation  task  would  have  been  more  diffieult  and  the  utility  of  RoboLeader  and  the  effeets 
of  operator  spatial  ability  might  have  been  even  more  pronouneed.  While  there  was  no 
signifieant  differenee  in  target  deteetion  performanee  among  the  eonditions,  the  participants 
reported  signifieantly  higher  workload  when  they  were  in  the  Manual  eondition  than  when  they 
were  assisted  by  RoboLeader.  In  eonclusion,  RoboLeader  demonstrated  signifieant  utility  for 
enhaneing  human  operators’  performanee  as  well  as  reducing  their  workload. 
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6.  Transitions 


RoboLeader  is  being  evaluated  as  an  intelligent  agent  for  the  Safe  Operations  for  Unmanned 
Vehicles  for  Reconnaissance  in  Complex  Environments  (SOURCE)  Army  Technology  Objective 
(ATO)  to  aid  operators  in  finding  safe  routes  for  manned  and  unmanned  systems  in  urban 
environments.  The  initial  study  found  important  individual  differences  related  to  perceived 
control  and  spatial  abilities  for  finding  safe  routes  and  identifying  targets  in  a  simulated  urban 
environment.  Furthermore,  the  participants  showed  interesting  effects  related  to  agent  reliability 
with  information  that  the  RoboLeader  failed  to  provide  (misses)  more  detrimental  to  finding  a 
safe  route  than  incorrect  information  (false  alarms).  This  study  is  reported  in  an  article  submitted 
to  the  6th  ACM/IEEE  International  Conference  on  Human-Robot  Interaction  {21).  Further 
experimentation,  evaluating  RoboLeader  as  decision  support  for  robotic  operators  is  planned  for 
2011  and  2012  as  part  of  the  SOURCE  ATO.  Finally,  the  RoboLeader  simulation  and  results  has 
been  chosen  as  one  of  the  demonstrations  for  the  final  project  of  the  NATO’s  Human  Factors  and 
Medicine  Panel  Working  Group  on  Supervisory  Control  (HFM-170).  The  demonstration  will  be 
documented  in  the  HFM-170  final  report  for  which  a  chapter  has  been  submitted  (22). 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ARL  U.S.  Army  Research  Laboratory 

ATO  Army  Technology  Objective 

DRI  Director’s  Research  Initiative 

MIX  Mixed  Initiative  Experimental 

NASA-TLX  National  Aeronautics  and  Space  Administration  Task  Load  Index 
OCU  Operator  Control  Unit 

PAC  perceived  attentional  control 

SA  situational  awareness 

SOURCE  Safe  Operations  for  Unmanned  Reconnaissance  in  Complex  Environments 

SpA  spatial  ability 

UAVs  unmanned  aerial  vehicles 

UGVs  unmanned  ground  vehicles 


UV  unmanned  vehicles 
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No.  of 

Copies  Organization 

1  ADMNSTR 

ELEC  DEFNS  TECHL  INFO  CTR 
ATTN  DTICOCP 

8725  JOHN  J  KINGMAN  RD  STE  0944 
FT  BEL  VOIR  VA  22060-6218 

1  CD  US  ARMY  RSRCH  LAB 

ATTN  RDRLCIMG  T  LANDFRIED 
BLDG  4600 

ABERDEEN  PROVING  GROUND  MD 
21005-5066 

3  CDS  US  ARMY  RSRCH  LAB 
ATTN  IMNEALCHRR 
MAIL  &  RECORDS  MGMT 
ATTN  RDRL  CIM  L  TECHL  LIB 
ATTN  RDRL  CIM  P  TECHL  PUB 
ADELPHI  MD  20783-1197 

I  US  ARMY  RSRCH  LAB 
ELEC  ATTN  RDRL  HRM  AT  J  CHEN 
12423  RESEARCH  PARKWAY 
ORLANDO  FL  32826 

TOTAL:  6  (2  ELEC,  4  CDS) 
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